PC-based computing system employing a multi-GPU graphics pipeline architecture supporting multiple modes of GPU parallelization dymamically controlled while running a graphics application

ABSTRACT

A PC-based computing system capable of displaying images of 3-D objects during an interactive process between said computing system and a user thereof. The PC-based computing system includes system memory for storing software graphics applications, software drivers and graphics libraries, and an operating system (OS), stored in the system memory, and a central processing unit (CPU), for executing the OS, graphics applications, drivers. and graphics libraries. The system also includes an CPU interface module and a PC bus, a graphics processing subsystem interfaced with the CPU interface module by way of the PC bus, and a display surface for displaying said images by graphically displaying frames of pixel data produced by the graphics processing subsystem. The graphics processing subsystem includes a plurality of GPUs arranged in a parallel architecture and operating according to a parallelization mode of operation so that each GPU supports a graphics pipeline and is allowed to process data in a parallel manner. A hardware hub, interfaces with the CPU interface module and the GPUs, by way of the PC bus, and has a hub router for (i) distributing the stream of geometrical data and graphic commands among the GPUs, and (ii) transferring pixel data output from one or more of GPUs during the composition of frames of pixel data corresponding to final images for display on the display surface. The CPU interface module provides an interface between one or more software hub drivers and the hardware hub.

RELATED CASES

The Present application is a Continuation of copending application Ser.No. 10/579,682 filed May 17, 2006, which is a National Stage Entry ofInternational Application No. PCT/IL2004/001069 filed Nov. 19, 2004,which is based on U.S. Provisional Application Nos. 60/523,084 filedNov. 19, 2003, and 60/523,102 filed Nov. 19, 2003; wherein each saidapplication is commonly owned by LUCID INFORMATION TECHNOLOGY LTD., andincorporated herein by reference in its entirety, as if set forth fullyherein.

BACKGROUND OF INVENTION Field of the Invention

The present invention relates to a method and system for 3-D(three-dimensional) multiple graphic processing. More particularly, theinvention relates to a method and system for improving theparallelization of image processing by Graphic Processing Units (GPUs),based on unified framework of three parallelization methods, which aretime division, image division and object division methods.

Definitions, Acronyms and Abbreviations

Throughout this patent Specification, the following definitions areemployed:

GPU: GPU (Graphic Processing Unit) like the CPU (Central ProcessingUnit), a GPU is a single-chip processor which is used primarily forcomputing 3-D functions. This includes tasks such as, lighting effects,object transformations, and 3-D motion. These aremathematically-intensive tasks, which otherwise, would put quite astrain on the CPU, but since the specialized GPU can handle most of the3-D calculations, it helps the computer to perform these tasks moreefficiently, and, of course, faster.

Polygon: Polygons in 3-D graphics are two-dimensional primitives,allowing generating and displaying of 3-D complex graphical objects. Thepolygons are the faces of the object and are composed from N vertices.Actually, a polygon is a closed plane figure, bounded by three or moreline segments.

Frame Buffer: a Frame Buffer (FB) is a buffer that stores the contentsof an image, pixel by pixel. Generally, the portion of memory isreserved for holding the complete bit-mapped image that is sent to themonitor, for display.

Typically the frame buffer is stored in the memory chips on the videoadapter. In some instances, however, the video chipset is integratedinto the motherboard design, and the frame buffer is stored in thegeneral main memory.

Object compositing unit: performs re-composition of multiplethree-dimensional rasters into final image. The merged data is resolvedfor the closest pixel to the viewer in 3-D space, based on the depthvalue of pixels. The new method, based on autonomous associativedecision, allows the use of multiple GPUs for any frame complexity.

Display list: a Display List is a description of the 3-D scene through alist of graphic primitives, such as polygons and attributes. The displaylist provides intermediate image storage for quick image retrieval.

Vertex array: a Vertex Array is an array of vertices describing the 3-Dscene.

A Vertex Array provides intermediate image storage for quick imageretrieval.

Alpha blending: Alpha blending controls the way in which the graphicinformation is displayed, such as levels of transparency, or opacity.

BRIEF DESCRIPTION OF THE STATE OF THE ART

The three-dimensional graphic pipeline architecture breaks-down intosegmented stages of CPU, Bus, GPU vertex processing and GPU fragment(pixel) processing. A given pipeline is only as strong as the weakestlink of one of the above stages, thus the main bottleneck determines theoverall throughput. Enhancing performance is all that required forreducing or eliminating bottlenecks. The major bottleneck stronglydepends on the application. Extreme cases are CAD-like (Computer AidedDesign) applications, characterized by an abundance of polygons(vertices), vs. video-game applications having a small polygon count butintensive fragment activity (e.g., texturing). The first class suffersfrom vertex processing bottlenecks, while the second class suffers fromfragment bottlenecks. Both are frequently jammed over the PC bus. Manyapplications have mixed characteristics, where bottlenecks may randomlyalternate between extremes, on a single frame basis.

The only way to improve the performance of the GPU is by means ofparallelizing multiple GPUs according to one of the bottleneck solvingmethods. There are two predominant methods for rendering graphic datawith multiple GPUs. These methods include time division (time domaincomposition), in which each GPU renders the next successive frame, andimage division (screen space composition), in which each GPU renders asubset of the pixels of each frame. The third one, much less popular, isthe object division (polygon decomposition) method.

In the time division method each GPU renders the next successive frame.It has the disadvantage of having each GPU render an entire frame. Thus,the speed at which each frame is rendered is limited to the renderingrate of a single GPU. While multiple GPUs enable a higher frame rate, adelay can be imparted in the response time (latency) of the system to auser's input. This occurs because, while at any given time, only one GPUis engaged in displaying a rendered frame, each of the GPUs is in theprocess of rendering one of a series of frames in a sequence. Tomaintain the high frame rate, the system delays the user's input untilthe specific GPU, which first received the signal cycles through thesequence, is again engaged in displaying its rendered frame. Inpractical applications, this condition serves to limit the number ofGPUs that are used in a system. With large data sets, there is anotherbottleneck, due to the fact that each GPU must be able to access all thedata. This requires either maintaining multiple copy operations of largedata sets or possible conflicts in accessing the single copy operation.

Image division method splits the screen between N GPUs, such that eachone displays 1/N of the image. The entire polygon set is transferred toeach GPU for processing, however, the pixel processing is significantlyreduced to the window size. Image division has no latency issues, but ithas a similar bottleneck with large data sets, since each GPU mustexamine the entire database to determine which graphic elements fallwithin the portion of the screen allocated to said GPU. Image divisionmethod suits applications with intensive pixel processing.

Object division method is based on distribution of data subsets betweenmultiple GPUs. The data subsets are rendered in the GPU pipeline, andconverted to Frame Buffer (FB) of fragments (sub-image pixels). Themultiple FB's sub-images have to be merged (composited) to generate thefinal image to be displayed. Object division delivers parallel renderingon the level of a single frame of very complex data consisting of largeamount of polygons. The input data is decomposed in the polygon leveland re-composed in the pixel level. A proprietary driver intelligentlydistributes data streams, which are generated by the application,between all GPUs. The rasters, generated by the GPUs, are compositedinto final raster, and moved to the display. The object division methodwell suits applications that need to render a vast amount of geometricaldata. Typically, these are CAD, Digital Content Creation, and comparablevisual simulation applications, considered as “viewers,” meaning thatthe data has been pre-designed such that their three-dimensionalpositions in space are not under the interactive control of the user.However, the user does have interactive control over the viewer'sposition, the direction of view, and the scale of the graphic data. Theuser also may have control over the selection of a subset of the dataand the method by which it is rendered. This includes manipulating theeffects of image lighting, coloration, transparency and other visualcharacteristics of the underlying data.

In above applications, the data tends to be very complex, as it usuallyconsists of massive amount of geometrical entities at the display listor vertex array.

Thus, the construction time of a single frame tends to be very long(e.g., typically 0.5 sec for 20 million polygons), which in turn slowsdown the overall system performance.

Therefore, there is a need to provide a system which can guarantee thebest system performance, being exposed to high traffic over the PC(Personal Computer) Bus.

OBJECTS OF THE PRESENT INVENTION

Accordingly, it is an object of the present invention to amplify thestrength of the GPU by means of parallelizing multiple GPUs.

It is another object of the present invention to provide a system,wherein the construction time of a single frame does not slow down theoverall system response.

It is still another object of the present invention to provide a systemand method, wherein the graphic pipeline bottlenecks of vertexprocessing and fragment processing are transparently and intelligentlyresolved.

It is still a further object of the present invention to provide asystem and method that has high scalability and unlimited scenecomplexity.

It is still a further object of the present invention to provide aprocess overcoming difficulties that are imposed by the datadecomposition, which is partition of data and graphic commands betweenGPUs.

It is still a further object of the present invention to provide amethod and system for an intelligent decomposition of data and graphiccommands, preserving the basic features of graphic libraries as statemachines and complying with graphic standards.

Other objects and advantages of the invention will become apparent asthe description proceeds.

SUMMARY OF THE INVENTION

The present invention is directed to a system for improving theparallelization of image processing, using one or more parallelizationmodes, wherein the image that is displayed on at least one computerscreen by one or more Graphic Processing Units, which comprises: one ormore software applications, for issuing graphic commands; one or moregraphic libraries, for storing data used to implement the graphiccommands; one or more Software Hub Drivers, for controlling a HardwareHub, for interacting with the operation system of the computer and thegraphic libraries, for performing real-time analysis of a data stream,from which frames of the image are generated, for determining theparallelization mode of each GPU, and for forwarding the data stream ora portion thereof to each GPU; one or more GPU Drivers, for allowing theGPUs to interact with the graphic libraries; and at least one I/O modulefor interconnecting between the Software module and the Hardware Hub,wherein, the Hardware Hub distributes between the GPUs, for each frame,graphic commands and the data stream or a portion thereof, according totheir relative complexity within the image, and defines the complexityThe Software Hub Driver also composites a graphics output for display,using the outputs obtained from at least one GPU, while alternating,whenever required, the parallelization mode for the each frame.

Parallelization is based on an object division mode or on an imagedivision mode or on a time division mode or on any combination thereof.The hardware hub comprises a compositing unit for composing a completeframe from processed portions of the data stream. The hardware hubcomprises a hub router for routing polygonal data, for routing graphiccommand stream, for routing pixel data and for routing the results ofcomposition, while operating in the object division mode or in the imagedivision mode or in the time division mode or in any combinationthereof. The hardware hub comprises a control unit for receivingcommands from the Software Hub Driver within the I/O module. Thehardware hub comprises a memory unit for storing intermediate processingresults of one or more GPUs and data required for composition andtransferring the processed data for display.

Preferably, the Software Hub Driver is capable of performing thefollowing operations: interception of the graphic commands from thestandard graphic library by means of the OS interface and utilities;forwarding and creating graphic commands to the GPU Driver by means ofthe OS interface and utilities; controlling the Hardware Hub, registryand installation operations by means of the OS interface and utilities:maintaining the consistency of graphic machine states across the GPUs,based on the input graphic commands stream, while using statemonitoring; estimating the type of graphic load and overload in theexecuted application graphic context, while using application andgraphic resources analysis; load estimation of the GPUs load balancebased on graphic commands stream and time measurements, while usingapplication and graphic resources analysis; adjusting the loaddistribution between GPUs according to feedback received from each GPUregarding the load balance, while using application and graphicresources analysis; performing manipulation in graphic functionsaccording to the current parallelization mode; and controlling thedistributed graphic functions, while modifying the graphic commands andthe data stream according to the current parallelization mode.

The present invention is directed to a method for improving theparallelization of image processing, using one or more parallelizationmodes, wherein the image that is displayed on at least one computerscreen by one or more Graphic Processing Units. Software applicationsare provided for issuing graphic command and graphic libraries areprovided for storing data used to implement the graphic commands. ASoftware Hub Drivers is provided for controlling a Hardware Hub, forinteracting with the operation system of the computer and the graphiclibraries, for performing real-time analysis of a data stream, fromwhich frames of the image are generated, for determining theparallelization mode of each GPU, and for forwarding the data stream ora portion thereof to each GPU. GPU Drivers are provided for allowing theGPUs to interact with the; graphic libraries and an I/O module isprovided for interconnecting between the Software module and theHardware Hub. Graphic commands and the data stream or a portion thereofare distributed between the GPUs for each frame by the Hardware Hub,according to their relative complexity within the image, wherein thecomplexity is defined by the Software Hub Driver. The Software HubDriver also composites a graphics output for display, using the outputsobtained from at least one GPU, while alternating, whenever required,the parallelization mode for the each frame.

Whenever the parallelization mode is an Object division parallelizationmode, the following steps are performed: for each frame, generating astream of graphic operations and polygonal data; marking the polygonaldata and graphic commands by means of the Software Hub Driver fordistribution between multiple GPUs; sending the marked data to theHardware Hub; distributing the marked data via the Hub Router to themultiple GPUs; rendering the data by means of GPUs; retrieving the datafrom the Frame Buffers and forwarding the retrieved data to thecompositing unit via the Hub Router; compositing the content of theFrame Buffers into a single Frame Buffer; and forwarding the content ofthe single Frame Buffer to at least one designated GPU for display.

Whenever the parallelization mode is an Image division parallelizationmode, the following steps are performed: subdividing the screen toportions and assigning different viewports to GPUs by means of theSoftware Hub Driver; moving the entire polygonal data and graphiccommands to the Hub Router; transmitting the entire polygonal data andgraphic commands to GPUs, wherein each GPU receives the same data;rendering the data by means of GPUs; forwarding a portion of the contentstored in the Frame Buffers to compositing unit in Hardware Hub for thecomplete image creation; and forwarding the image to at least onedesignated GPU for display.

Whenever the parallelization mode is a Time division parallelizationmode, the following steps are performed: forwarding to each one of themultiple GPUs the entire amount of polygons for rendering; redirectingthe entire polygonal data and graphic commands by means of Software HubDriver to all GPUs, while alternating between them; rendering the databy means of GPUs; transferring rendered data from at least one GPU viathe Hub Router; and redirecting the resulting content of the FrameBuffer via Hub Router to at least one designated GPU for display.

The distribution of polygons between multiple GPUs is performed bydistributing blocks of data between multiple GPUs and by testing eachgraphic operation for blocking mode, in which one or moreparallelization modes are carried out, thereafter. The data isredirected in regular non-blocking path to at least one designated GPU,This process is repeated until a blocking operation is detected. ThenGPUs are synchronized by performing a flush operation in order toterminate rendering and clean up the internal pipeline in each GPU;performing a composition operation for merging the contents of the FrameBuffers into a single Frame Buffer and by transmitting the single FrameBuffer back to all GPUs. Then the composited complete frame isterminated at all GPUs, except one or more designated GPUs, whenever aSwap operation is detected and displaying the image by means of the oneor more designated GPUs. the same data is processed by all GPUs, as longas the blocking mode is active and the Swap operation is not detected.Whenever the blocking mode is inactive, the designated data is furtherprocessed by multiple GPUs.

BRIEF DESCRIPTION OF THE DRAWINGS

For a more complete understanding of the present invention, thefollowing Detailed Description of the Illustrative Embodiment should beread in conjunction with the accompanying Drawings, wherein:

FIG. 1 is a block diagram of a multiple GPU architecture system,according to an embodiment of the present invention;

FIG. 2 is a block diagram of Hardware Hub components, according to anembodiment of the present invention;

FIG. 3 is a block diagram of Object division parallelization mode,according to an embodiment of the present invention;

FIG. 4 is a block diagram of Image division parallelization mode,according to an embodiment of the present invention;

FIG. 5 is a block diagram of Time division parallelization mode,according to an embodiment of the present invention;

FIG. 6 is a schematic block diagram of a possible integration of theSoftware Hub Driver into the operating system environment, according toan embodiment of the present invention;

FIG. 7 is a functional block diagram presenting the main tasks of theSoftware Hub Driver, according to an embodiment of the presentinvention;

FIG. 8 is a flow chart presenting an process for distribution of thepolygons between the multiple GPUs, according to an embodiment of thepresent invention; and

FIG. 9 discloses a sample configuration of the system, employing 8 GPUs,according to an embodiment of the present invention.

It will be appreciated that for simplicity and clarity of illustration,elements shown in the figures have not necessarily been drawn to scale.For example, the dimensions of some of the elements may be exaggeratedrelative to other elements for clarity. Further, where consideredappropriate, reference numerals may be repeated between the figures toindicate corresponding or analogous elements.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The Multiple 3-DGraphic Pipeline

The current invention calls for the introduction of an extended PCgraphic architecture including novel operational component, the 3-Dpipeline Hub.

FIG. 1 presents multiple GPU architecture system 100, according to anembodiment of the present invention. The hub 110 is located in a uniqueposition on the PC bus, between the CPU (Central Processing Unit) and acluster of GPUs 130. The multiple graphic pipeline architecture, asdescribed in FIG. 1, consists of Software Application 121, GraphicLibrary 122, Software Hub Driver 123, GPU Driver 124, Hardware Hub 110,cluster of GPUs 130, and display (s) 140. Usually, one of the GPUs isdesignated as a display unit. It should be noted, that it is possible tohave more than one display unit, or include a display unit directlyinside the Hardware Hub. A display unit can drive multiple screens, aswell.

The Hub mechanism consists of a Hardware Hub component 110, located onthe PC bus between I/O (Input/Output) chipset 160, being a I/O module,and a cluster of GPUs 130, and a Software module comprising Software HubDriver 123, executed by the PC.

The Hardware Hub 110 carries on at least the following action:distributes decomposed polygon stream and graphic commands between GPUs;composites graphics output for display according to different parallelmodes; makes cache of data; and alternates modes of parallelism.

The Software Hub Driver 123, besides controlling the Hardware Hub 110,also carries on at least the following actions: interacts with the OS(Operation System) and graphic library, such as OpenGL, DirectX;performs real-time analysis of the data stream; determines theparallelization mode; and—decomposes the data and command stream.

One advantage of this method is that the unique location of the HardwareHub 110 allows it to control the graphic pipeline, while beingtransparent to the application 121. The application 121, along withGraphic Library 122, such as OpenGL, keeps working as it was a singleGPU.

Another advantage of this method is that the unique location of theHardware Hub 110 allows it to control a graphic pipeline between theUser Interface 150 and Display 140, while being transparent to the GPU.Each GPU of the cluster keeps working as if it is the only graphicprocessor hooked on the I/O chipset 160.

Still another advantage of this method is that the unique location ofthe Hardware Hub 110 allows it to control the graphic pipeline fordifferent parallelization modes: image division mode, time division modeor object division mode.

A further advantage of this method is that the unique location of theHardware Hub 110 allows it to sense in real-time the varying parametersof application's load, such as polygon count, texture volume, humaninteraction, and to intelligently decide and carry on the currentoptimal parallelization method.

It should be noted that according to all embodiments of the presentinvention, the display(s) 140 may be connected directly or indirectly tothe Hardware Hub 110.

Reference is now made to FIG. 2, which discloses the components of theHardware Hub 110, according to an embodiment the present invention.Control Unit 210, accepts proprietary commands from the Software HubDriver over the bus, and accordingly controls the other units. HubRouter 230 routs polygonal data and graphic command stream from left toright, routs pixel data from right to compositing unit, and routscompositing results to the right. Compositing Unit 240 performs variouscompositing schemes according to parallelization mode. Auxiliary Memoryblock 220 is used for storing intermediate processing results of one ormore GPUs, data required for composition and transferring the processeddata for display.

The Hardware Hub 110 utilizes its units according to parallelizationmodes: an Object division mode, an Image division mode, a Time divisionmode. These modes are adaptively handled by the Hardware Hub 110according to application needs.

FIG. 3 discloses the operation of the Object division parallelizationmode, according to an embodiment the present invention. The CPU executesthe 3-D graphic application 310, which along with standard graphiclibrary, generates a stream of graphic operations and polygonal data.They are typically organized in data blocks either as Display List,Vertex Array or free polygons, which are polygons that are neitherorganized in Display List nor in Vertex Array. The Software Hub Driverat step 320 marks the polygonal data and graphic commands fordistribution between multiple GPUs, in a way that the overall load isbalanced. The marked data is forwarded to Hardware Hub. At step 330 itis distributed via the Hub Router to multiple GPUs. After the renderingprocess in GPUs at step 340 is accomplished, the Frame Buffers areretrieved and forwarded via the Hub Router to the compositing unit atstep 350. Here the frame buffers are composited to a single framebuffer, which is forwarded to the designated GPU for display. The singleframe buffer is displayed at step 360.

FIG. 4 discloses the operation of the Image division parallelizationmode, according to an embodiment the present invention. In this mode theSoftware Hub Driver assigns different viewports to GPUs in order tosubdivide the screen between them. The viewports aspects are setaccording to load balancing considerations, to keep the overall GPU loadevenly balanced. In step 420, the entire polygonal data and graphiccommands are moved to the Hub Router at Hardware Hub, and they aretransmitted at step 430 to multiple GPUs. All GPUs receive the samedata. After rendering at step 440, the partial frame buffers are broughtto compositing unit in Hardware Hub for the full image creation at step450, and then this image is moved out to designated GPU for display. Thefull image is displayed at step 460.

FIG. 5 discloses the operation of the Time division parallelizationmode, according to an embodiment the present invention. In time divisionthe processing of each frame takes N frame time units, while N GPUs (orN clusters of GPUs) are participating. The entire amount of polygons isforwarded to each GPU, for rendering. At each frame time unit, theSoftware Hub Driver redirects the polygonal data and graphic commands atstep 530 to a cluster of GPUs at a time, while alternating between them.The data is transferred to the above cluster of GPUs via the Hub Router,rendered in the GPUs at step 540, and then the resulting frame buffer atstep 550 is redirected via Hub Router to the designated GPU for display.All GPUs are coordinated by Software Hub Driver to create a continuoussequence of frames. The resulting frame buffer is displayed at step 560.

The Hardware Hub competence is its scaling technology: Architecture andcluster of proprietary processes devoted to scale existing GPUsperformance in PC based systems, by enabling the use of multiple GPUs inparallel on the level of chip, card or chip IP (Intellectual Property)core, and handling multiple bus paths between the CPU and GPU. Thetechnology achieves linear increase in performance. It is invariant to agraphics vendor and also it is transparent to an application. In thepresent invention, the graphic pipeline bottlenecks of vertexprocessing, fragment processing and bus transfer are completely andintelligently resolved. As bottlenecks may shift between frames, theHardware Hub is designed with a smart real-time feedback system betweenthe Control Unit 210, disclosed in FIG. 2, and Software Hub Driver 123,disclosed in FIG. 1, by means of the bus, utilizing the differentparallelization modes to overcome different bottlenecks and maintainmaximum performance at the frame level.

The Software Hub Driver

The Software Hub Driver is a software package residing in the PC andcoexisting with computer's operating system, standard graphic library,application and Vendor's GPU Driver. FIG. 6 is a schematic block diagramof a possible integration of the Software Hub Driver 630 into theoperating system environment according to an embodiment the presentinvention. Next to graphic application block 610 there is standardgraphic library block 620. The Software Hub Driver 630 is locatedbeneath the standard graphic library 620, intercepting the graphiccommand and data stream on the way to the Vendor's GPU Driver 640. TheSoftware Hub Driver 630 also controls the Hardware Hub 660.

FIG. 7 is a functional block diagram presenting the main tasks of theSoftware Hub Driver, according to an embodiment the present invention.OS interface and Utilities block 710 is responsible for interception ofthe graphic commands from the standard graphic library, forwarding andcreating graphic commands to Vendor's GPU Driver, controlling theHardware Hub, registry and installation, OS services and utilities.State Monitoring block 720 is responsible for maintaining consistency ofgraphic machine states across the GPUs, based on the input graphiccommands stream. Application and graphic resources analysis block 730 isresponsible for the application observation-estimating the type ofgraphic load and bottleneck in the current application graphic context,graphic resources (GPUs) load estimation for load balance based ongraphic commands stream and time measurements, handling the feedbackfrom GPUs in regard to load balancing. Parallelism policy managementblock 740 is based on load analysis. All parallelization modes, whichare the Object division mode, Image division mode and Time divisionmode, are combined together in order to achieve best performance andoptimal load handling. Parallelization policy is based on the analysisof the load, and it must preserve the state of the graphic system at allrelevant GPUs across the electronic circuit or chip. For example,changing of a state by adding a new light source in the scene at sometime point, must affect all subsequent polygons at different GPUs.Parallelism policy management block 740 is responsible for theinterpretation of the policy for specific manipulation in graphicfunctions. Distributed graphic functions control block 750 isresponsible for modification of graphic command and data stream based onthe parallelization policy.

Object Division Decomposition Process

Object division is a well-known concept, but data decomposition(partition of data and graphic commands between GPUs), while being alsoa known concept, has not been applied yet effectively, as it imposesvarious great difficulties. These difficulties are handled successfullyby a proposed process and its implementation, according to the presentinvention.

The decomposition, and more importantly, the composition, must beaccurate and efficient. Certain operations must be performed in theorder they are submitted by the application. For example, in case ofsemi-transparency, the commands and polygon stream must keep a certainorder for creating a correct graphic result.

Intelligent decomposition of data and graphic commands is needed,preserving the basic features of graphic libraries as state machines,and complying with the graphic standards. The proposed decompositionprocess, according to the present invention, is performed by theSoftware Hub Driver. CPU runs the 3-D graphic application, generatingflow of graphic commands and data. They are typically organized inblocks, such as Display Lists or Vertex Arrays, stored in the systemmemory.

According to the present invention, the Software Hub Driver, running inthe CPU, decomposes the set of scene polygons (or vertices). Theirphysical distribution is performed by the Hardware Hub.

The polygons are rendered in the GPU, while maintaining the resultingFrame Buffer in local memory. All FBs are transferred, via Hub Router,to compositing unit in Hardware Hub, to be merged into single FB.Finally, the composited FB is forwarded for display.

The Software Hub Driver carries out the following process ofdistribution of the polygons between the multiple GPUs. It is assumed,that the regular way the graphic application works, remains unchanged.Per frame, a typical application generates a stream of graphic callsthat includes blocks of graphic data; each block consists of a list ofgeometric operations, such as single vertex operations or buffer basedoperations (vertex array). Typically, the decomposition process splitsthe data between GPUs preserving the blocks as basic data units.Geometric operations are attached to the block(s) of data, instructingthe way the data is handled. A block is directed to designated GPUs.However, there are operations belonging to the group of BlockingOperations, such as Flush, Swap, Alpha blending, which affect the entiregraphic system, setting the system to blocking mode. Blocking operationsare exceptional in that they require a composed valid FB data, thus inthe parallel setting of the present invention, they have an effect onall GPUs. Therefore, whenever one of the Blocking operations is issued,all the GPUs must be synchronized. Each frame has at least 2 blockingoperations: Flush and Swap, which terminate the frame.

FIG. 8 is a flow chart presenting the process for distribution of thepolygons between the multiple GPUs, according to an embodiment thepresent invention. The frame activity starts with distributing blocks ofdata between GPUs. Each graphic operation is tested for blocking mode atstep 820. In a regular (non-blocking) path, data is redirected to thedesignated GPU at the step 830. This loop is repeated until a blockingoperation is detected.

When the blocking operation is detected, all GPUs must be synchronizedat step 840 by at least the following sequence: —performing a flushoperation in order to terminate rendering and clean up the internalpipeline (flushing) in GPU; —performing a composition in order to mergethe contents of FBs into a single FB; and —transmitting the contents ofsaid single FB back to all GPUs, in order to create a common ground forcontinuation.

The Swap operation activates the double buffering mechanism, swappingthe back and front color buffers. If Swap is detected at step 850, itmeans that a composited complete frame must be terminated at all GPU,except GPU0. All GPUs have the final composed contents of a FBdesignated to store said contents, but only the one connected to thescreen (GPUO) displays the image at step 860.

Another case is operations that are applied globally to the scene andneed to be broadcasted to all the GPUs. If one of the other blockingoperations is identified, such as Alpha blending for transparency, thenall GPUs are flushed as before at step 840, and merged into a common FB.This time the Swap operation is not detected (step 850), and thereforeall GPUs have the same data, and as long as the blocking mode is on(step 870), all of them keep processing the same data (step 880). If theend of the block mode is detected at step 870, GPUs return working ondesignated data (step 830).

Adaptive Handling of Graphic Load by Combining Three Division Methods

In addition, the present invention introduces a dynamic load-balancingtechnique that combines the object division method with the imagedivision and time division methods in image and time domains, based onthe load exhibits by previous processing stages. Combining all the threeparallel methods into a unified framework dramatically increases theeffectiveness of our invention.

Parallel processing is implemented by a pipeline, such as any common GPUallows the data to be processed in parallel, in time, image and objectdomains. The processing performed on the graphical processing system,either in parallel on multi-GPU or sequential, results in a sequence ofcomplete raster images stored in a frame buffer, and sent to the displayunit. These images are referred as frames in short. A frame consists offragments. A fragment is an extended pixel stored in memory, whichconsists of attributes such as color, alpha, depth, stencil, etc. Whenprocessing is performed in parallel in the time domain, typically eachGPU is responsible for the production of a complete frame. In the othertwo domains, which are the image and object domains, all GPU operate inparallel to produce a single frame. Screen-space parallel-processingimplies that each GPU renders a subset of the fragments of each frame,and object parallel-processing implies that the input data for eachframe, in particular the geometric data (e.g., the polygon setrepresenting the scene) is distributed between the multiple GPUs.

Each one of three domains (time, image and object domains) hasadvantages and disadvantages. The effectiveness of each discipline is adynamic function based on input data. Moreover, in many cases no singlediscipline is superior. In these cases a combination of two or even allthe three disciplines may yield the most optimum results.

The present invention provides a parallel-processing system forthree-dimensional data. It provides a novel process for objectparallel-processing that consists of efficient decomposition of the databetween the different GPU, and then the composition of the framesproduced on the various GPUs into a final frame ready to be rendered.

The present invention provides a method to integrate all the threeparallel modes dynamically into a unified framework to achieve maximumload balancing. At each frame, the set of available GPUs can bereconfigured based on the time it took to render the previous frames,and the bottlenecks exhibited during the processing of these frames.

FIG. 9 discloses a sample configuration of the system, employing eight(8) GPUs, according to an embodiment of the present invention. Accordingto the above sample configuration, a balanced graphic application isassumed. The GPUs are divided into two groups for time divisionparallelism. GPUs indexed with 1, 2, 3, and 4 are configured to processeven frames and GPUs indexed with 5, 6, 7, and 8 are configured toprocess odd frames. Within each group, two GPU subgroups are set forimage division: the GPUs with the lower indexes (1,2 and 5,6respectively) are configured to process half of the screen, and thehigh-indexed GPU (3,4 and 7,8 respectively) are configured to processthe other half. Finally, for the object division, GPUs indexed with 1,3,5 and 7 are fed with half of the objects, and GPUs indexed with 2,4, 6and 8 are fed with the other half of the objects.

If at some point the system detects that the bottlenecks exhibited inprevious frames occur at the raster stage of the pipeline, it means thatfragment processing dominates the time it takes to render the frames andthat the configuration is imbalanced. At that point the GPUs arereconfigured, so that each GPU will render a quarter of the screenwithin the respective frame. The original partition for time division,between GPUs 1, 2, 3,4 and between 5,6, 7, 8 still holds, but GPU 2 andGPU 5 are configured to render the first quarter of screen in even andodd frames respectively. GPUs 1 and GPU 6—the second quarter, GPU 4 andGPU 7—the third quarter, and GPU 3 and GPU 8—the forth quarter. Noobject division is implied.

In addition, if at some point the system detects that the bottleneckexhibited in previous frames occurs at the geometry stage of the pipe,the GPUs are reconfigured, so that each GPU will process a quarter ofthe geometrical data within the respective frame. That is, GPU 3 and GPU5 are configured to process the first quarter of the polygons in evenand odd frames respectively. GPU 1 and GPU 7—the second quarter, GPU 4and GPU 6—the third quarter and GPU 2 and GPU 8—the forth quarter. Noimage division is implied.

It should be noted, that taking 8 GPUs is sufficient in order to combineall three parallel modes, which are time, image and object divisionmodes, per frame. Taking the number of GPUs larger than 8, also enablescombining all 3 modes, but in a non-symmetric fashion. The flexibilityalso exists in frame count in a time division cycle. In the aboveexample, the cluster of 8 GPUs was broken down into the two groups, eachgroup handling a frame. However, it is possible to extend the number offrames in a time division mode to a sequence, which is longer than 2frames, for example 3 or 4 frames.

Taking a smaller number of GPUs still allows the combination of theparallel modes, however the combination of two modes only. For example,taking only 4 GPUs enables to combine image and object division modes,without time division mode. It is clearly understood from FIG. 9, whiletaking the group of GPU1, GPU2, GPU3 and GPU4, which is the leftcluster. Similarly, the group of GPU1, GPU2, GPU5 and GPU6, which is theupper cluster, employs both object and time division modes. Finally, theconfiguration of the group of GPU2, GPU4, GPU5 and GPU6, which is themiddle cluster, employs image and time division modes.

It should be noted, that similarly to the above embodiments, anycombination between the parallel modes can be scheduled to evenlybalance the graphic load.

It also should be noted, that according to the present invention, theparallelization process between all GPUs may be based on an objectdivision mode or image division mode or time division mode or anycombination thereof in order to optimize the processing performance ofeach frame.

While some embodiments of the invention have been described by way ofillustration, it will be apparent that the invention can be put intopractice with many modifications, variations and adaptations, and withthe use of numerous equivalents or alternative solutions that are withinthe scope of persons skilled in the art, without departing from thespirit of the invention or exceeding the scope of the claims.

1. A PC-based computing system comprising: a system memory for storingsoftware graphics applications, software drivers and graphics libraries,and; an operating system (OS), stored in said system memory; one or moregraphics applications, stored in said system memory, for generating astream of geometrical data and graphics commands supporting (i) therepresentation of one or more 3D objects in a scene having 3Dgeometrical characteristics and (ii) the viewing of images of said oneor more 3D objects in said scene during an interactive process carriedout between said PC-based computing system and a user of said PC-basedcomputing system; one or more graphic libraries, stored in said systemmemory, for storing data used to implement said stream of geometricaldata and graphics commands; a central processing unit (CPU), forexecuting said OS, said graphics applications, said drivers and saidgraphics libraries; an CPU interface module for interfacing with saidCPU; a PC bus; a graphics processing subsystem interfaced with said CPUinterface module by way of said PC bus; and a display surface fordisplaying said images by graphically displaying frames of pixel dataproduced by said graphics processing subsystem; wherein said graphicsprocessing subsystem includes: a plurality of graphic processing units(GPUs) arranged in a parallel architecture and operating according toone or more parallelization modes of operation so that said GPUs supportmultiple graphics pipelines and process data in a parallel manner,wherein said one or more parallelization modes of operation include (i)a time division mode wherein each GPU renders a different frame of pixeldata to be displayed at a different moment of time, (ii) an imagedivision mode wherein each GPU renders a subset of the pixels used tocompose each frame of pixel data to be displayed, and (iii) an objectdivision mode wherein the object which is to be displayed as a frame ofpixels, is decomposed into said stream of geometrical data and graphiccommands which are distributed to said GPUs for rendering the frames ofpixel data compositing the images for display on said display surface;one or more GPU drivers, stored in said system memory, for allowing saidGPUs to interact with said graphic libraries; one or more software hubdrivers, stored in said system memory, a hardware hub, interfacing withsaid CPU interface module and said GPUs, by way of said PC bus, andhaving a hub router for (i) distributing the stream of geometrical dataand graphic commands among said GPUs, and (ii) transferring pixel dataoutput from one or more of said GPUs during the composition of frames ofpixel data corresponding to final images for display on said displaysurface; wherein said CPU interface module provides an interface betweensaid one or more software hub drivers and said hardware hub; whereinsaid one or more software hub drivers perform the following functions:(i) controlling the operation of said hardware hub, (ii) interactingwith said OS and said graphic libraries, (iii) analyzing said stream ofgeometrical data and said graphics commands, (iv) determining the modeof parallelization of said GPUs during each frame, using results of theanalysis of said stream of geometrical data and said graphics commands,and one or more parallelization policies, and (v) forwarding said streamof geometrical data and graphic commands, or a portion thereof, to eachsaid GPU over said PC bus; and wherein, for each image of said 3D objectto be generated and displayed on said display surface, the followingoperations are performed: (i) said hardware hub uses said hub router andsaid PC bus to distribute said stream of geometrical data and graphiccommands, or a portion thereof, to said GPUs, (ii) one or more of saidGPUs process said stream of geometrical data and graphic commands, or aportion thereof, during the generation of each said frame, whileoperating in said parallelization mode, so as to generate pixel datacorresponding to at least a portion of said image, and (iii) saidhardware hub uses said router and said PC bus to transfer said pixeldata output from one or more of said GPUs and compose a frame of pixeldata, representative of the image of said 3D object, for display on saiddisplay surface.
 2. The PC-based computing system of claim 1, whereinsaid CPU interface module comprises an I/O module.
 3. The PC-basedcomputing system of claim 2, wherein said I/O module comprises an I/Ochip or chipset.
 4. The PC-based computing system of claim 1, whereineach said GPU has a frame buffer (FB) for storing a fragment of pixeldata.
 5. The PC-based computing system of claim 1, wherein saidgeometrical data comprises a set of scene polygons or vertices.
 6. ThePC-based computing system of claim 1, wherein said graphics commandsincludes commands selected from the group consisting of display listsand display vertex arrays.
 7. The PC-based computing system of claim 1,wherein said hardware hub further comprises a control unit for acceptingcommands from said one or more software hub drivers, over said PC bus,and controlling components within said hardware hub, including said hubrouter.
 8. The PC-based computing system of claim 1, wherein said hubrouter routes said stream of geometrical data and graphic commands fromsaid graphics application to one or more of said GPUs, and wherein saidhub router routes pixel data results from said GPUs during thecomposition of said frame of pixel data.
 9. The PC-based computingsystem of claim 1, wherein said hardware hub further comprises a memoryunit for storing intermediate processing results from one or more ofsaid multiple GPUs and data required for composition and transferringframes of pixel data for display.
 10. The PC-based computing system ofclaim 1, wherein said one or more software hub drivers control said GPUswhile said hardware hub operates transparently to said graphicsapplication so that said multiple GPUs appear as only a single GPU tosaid graphics application.
 11. The PC-based computing system of claim 1,wherein said one or more software hub drivers coordinate the operationof said GPUs so generate a continuous sequence of frames of pixel datafor displaying a sequence of images of said 3D object on said displaysurface.
 12. The PC-based computing system of claim 1, wherein saidhardware hub handles multiple bus paths between said CPU and said GPUs.13. The PC-based computing system of claim 1, wherein saidparallelization mode of operation is a time division mode of paralleloperation, wherein each said GPU renders a different frame of pixel datafor display on said display surface at a different moment of time. 14.The PC-based computing system of claim 1, wherein said parallelizationmode of operation is an image division mode of parallel operation,wherein each GPU renders a subset of the pixels used to compose eachframe of pixel data to be displayed on said display surface.
 15. ThePC-based computing system of claim 1, wherein said parallelization modeof operation is an object division mode of parallel operation, whereinthe 3D object which is to be displayed as an image consisting of a frameof pixels, is decomposed into said stream of geometrical data andgraphic commands which are distributed to said GPUs for rendering theframes of pixel data compositing the images to be displayed on saiddisplay surface.
 16. The PC-based computing system of claim 1, whereinsaid determining the mode of parallelization of said GPUs during aparticular frame of pixel data generation and display is based onfactors including the time required to render previous frames of pixeldata and bottlenecks exhibited in vertex processing and pixel processingduring the rendering of said previous frames of pixel data.
 17. ThePC-based computing system of claim 1, wherein said parallelization modeoperation is based on any combination of said object division mode, saidimage division mode and said time division mode.
 18. The PC-basedcomputing system of claim 1, wherein the Hardware Hub comprises at leastone compositing unit at least for composing a complete frame of pixeldata from processed portions of geometrical data stream.
 19. ThePC-based computing system of claim 1, wherein said one or more softwarehub drivers is capable of performing operations selected from the groupconsisting of: (1) interception of graphic commands from said graphiclibraries by means of said OS and utilities; (2) forwarding and creatinggraphic commands to said GPU driver by means of said OS and utilities;and (3) controlling said hardware hub, registry and installationoperations by means of said OS interface and utilities: (4) maintainingthe consistency of graphic machine states across said GPUs, based onsaid stream of graphic commands, while using state monitoring; (5)estimating the type of graphic load and overload in the graphic contextof the executed graphics application, while using application andgraphic resources analysis; (6) estimation of the load balance of saidGPUs based on time measurements and said stream of geometrical data andgraphic commands while using application and graphic resources analysis;(7) adjusting the load distribution between said GPUs according tofeedback received from each GPU regarding said load balance, while usingapplication and graphic resources analysis; (8) performing manipulationof graphic functions performed by said GPUs, according to the currentparallelization mode; and (9) controlling the distributed graphicfunctions performed by said GPUs, while modifying said stream ofgeometrical data and graphic commands according to said currentparallelization mode.
 20. The PC-based computing system of claim 1,wherein each said 3D object is decomposable into a plurality ofpolygons, and wherein said geometrical data comprises the vertices ofsaid polygons.
 21. The PC-based computing system of claim 1, whereineach pixel associated with a frame of pixel data includes attributesselected from the group consisting of color, alpha, position, depth, andstencil.